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Abstract 


Many  critical  Command  and  Control  decisions,  such  as  Time- Sensitive  Targeting,  must 
be  made  collaboratively  due  to  the  amount  of  information  to  be  processed  and  the  level  of 
complexity.  This  research  addresses  the  challenges  of  technology- supported  human 
collaboration.  We  have  conducted  a  blend  of  field  observation,  expert  interviews,  and 
chat  analysis  at  several  joint,  time-sensitive  exercises  and  experiments. 

We  find  operator  teams  perform  critical  functions  as  part  of  the  human-technology 
“cognitive  functional  system.”  For  example,  they: 

•  validate  information,  and  determine  where  to  get  more  if  needed 

•  engage  in  collaborative  sensemaking  -  handling  ambiguous  or  conflicting 
information 

•  establish  tmst  and  credibility  with  one  another 

•  maintain  sufficient  team  awareness  to  enable  effective  coordination,  even 
when  not  co-located 

•  judge  who  should  -  and  should  not  -  receive  information,  balancing  the 
need  for  sharing  against  the  danger  of  cognitive  overload 

•  select  appropriate  communication  modalities  for  sharing  information  of 
varying  importance,  time-sensitivity,  and  intended  audience 

•  cue  other  team  members  to  important  information,  emerging  events,  or 
changing  priorities. 


Our  results  reveal  how  challenging  these  functions  can  be  for  operators  bombarded  with 
information,  how  existing  technology  supports  or  hinders  key  activities,  and  how  teams 
adapt  their  processes  or  technologies  to  meet  real  time  demands. 


INTRODUCTION 


Despite  the  many  advances  in  technology,  human  collaboration  and  team  decision 
making  remain  challenging  and  time-consuming  activities  in  complex  military  domains. 
In  fact,  a  recent  study  of  Time  Sensitive  Targeting  (TST)  processes  during  Operation 
Enduring  Freedom  concluded  that  “Complex  decision  making  processes  consume  a  far 
greater  proportion  of  the  TST  timeline  than  do  communications  between  sensors, 
shooters,  and  other  TST  process  components.”  The  report  articulated  the  need  for  “an 
assessment  mechanism  that  allows  for  the  human  factors  involved  in  decision  making 
when  complex  issues  are  involved”  (Veridian,  2003). 

The  goal  of  our  research  was  to  study  teams  of  operators  making  decisions  in 
realistic  time-sensitive  venues.  We  sought  to  understand  the  kinds  of  tasks  and  functions 
people  perform  in  these  collaborative  environments,  why  the  tasks  are  so  challenging, 
how  they  are  currently  done,  and  how  to  better  support  team  performance  through 
improved  technology  and  processes. 

In  military  domains  such  as  Time  Sensitive  Targeting,  both  the  particular 
technologies  used  and  the  individuals  involved  in  collaborative  decision  making  are 
constantly  changing.  Therefore,  instead  of  researching  how  operators  interact  with  one 
particular  technology,  or  how  one  specific  decision  is  made,  we  chose  to  observe  and 
analyze  the  broader  team  decision  making  process  as  it  unfolds  in  situ.  This  more 
ambitious  objective  meant  we  had  to  find  a  means  to  address  the  multiple  tools  and 
technologies  that  operator  teams  use  to  inform  their  judgments  -  to  include  domain 
specific  applications  tailored  to  particular  operator  roles,  more  generic  technologies  such 
as  instant  messaging  and  email,  face  to  face  conversations,  large  wall  displays,  and  phone 
calls.  Key  questions  we  sought  to  answer  include, 

•  What  are  the  fundamental  activities,  patterns,  and  challenges  in  the 
collaborative  process? 

•  Where  does  technology  support  or  inadvertently  hinder  decision 
making,  and  why? 

•  Do  available  technologies  work  together  in  a  complementary 
fashion,  or  are  they  difficult  to  use  in  combination? 

•  Are  the  sources  of  needed  information  -  whether  machine  or 
human  based  -  transparent  to  operators  and  easily  accessible? 


DATA  ON  TECHNOLOGY-SUPPORTED  TEAM  DECISION  MAKING 

To  adequately  capture  the  interactions  of  people,  processes,  and  technology  in 
context,  we  developed  a  coordinated  blend  of  three  approaches:  direct  field  observation 
of  operators,  expert  interviews,  and  analysis  of  instant  messaging  or  “chat”  logs. 

Direct  Observation 

In  direct  field  observation,  we  serve  as  unobtrusive  observers  and  are  co-present 
with  operators  in  their  normal  work  environment.  This  is  consistent  with  ethnographic 
observation  (Nardi  and  O’Day,  1999)  and  contextual  inquiry  techniques  (Holtzblatt  and 


Jones,  1993).  We  had  successfully  used  observation  techniques  in  previous  military 
environments  (e.g.,  Swanson,  Drury,  and  Lewis,  2004).  Over  2004,  we  gained  access  to 
several  Time  Sensitive  Targeting-related  exercises  and  experiments  that  closely 
resembled  the  actual  operational  work  environment.  With  permission  from  the  operators, 
we  placed  ourselves  near  enough  to  hear  conversations,  see  tool  placement  and  usage, 
observe  when  operators  phoned  or  got  up  to  speak  to  others,  and  detect  more  subtle  body 
language  suggesting  frustration,  confusion,  fatigue,  or  other  mental  and  emotional  states. 
We  looked  over  their  shoulders  in  order  to  document  which  applications  they  used,  and 
how.  In  some  cases,  operators  even  invited  us  to  listen  in  via  dual  headsets  to  hear  those 
communications  in  real  time. 

Direct  observation  is  valuable  for  monitoring  the  full  communication  spectrum, 
from  individuals  typing  into  chat  windows  to  face-to-face  discussions  and  phone  calls.  It 
also  enables  detection  and  documentation  of  physical  cues  (facial  expressions,  joking, 
banging  on  the  table,  etc.)  that  can  reveal  emotions  or  cognitive  states.  Another  benefit 
of  field  observation  is  the  immediacy  of  the  data;  team  interactions  and  activities  are 
captured  as  quickly  as  we  can  note  them  on  paper  so  the  latency  relative  to  actual  events 
is  negligible.  A  primary  disadvantage  of  directly  observing  operators  in  the  field, 
however,  is  that  a  few  observers  cannot  possibly  attend  to  all  team  members  and  all 
activities  at  once;  it  is  very  difficult  to  capture  an  entire  collaborative  thread  using  this 
approach,  especially  if  the  team  is  not  fully  co-located.  In  classified  environments,  audio 
or  visual  recordings  are  not  permitted  so  field  observations  must  be  recorded  by  hand. 


Data  Collection  Methodology 

Direct  Observation  (“actions”) 
Chat  Logs  (“words”) 


Expert  Interviews  (“thoughts”) 


Figure  1.  Data  for  Analyzing  Technology- Supported  Team  Decision  Making 


Expert  Interviews 

We  also  gathered  data  via  structured  interviews.  We  focused  on  operators  with 
extensive  experience  in  the  field  who  could  provide  insights  based  on  their  years  of 
domain  experience.  During  interviews,  we  solicit  their  rationales,  their  judgments  on  the 
amount  and  quality  of  information  provided,  their  success  stories  or  perceived  challenges 
involving  information  sharing  and  coordination,  and  their  perspectives  on  social  issues 
such  as  the  level  of  team  familiarity,  cohesiveness,  and  trust.  It  is  critical  that  questions 
do  not  annoy  or  distract  team  members  from  their  activities.  Questions  are  typically 
asked  before  or  after  an  exercise,  or  during  a  break  in  the  action  if  an  operator  indicates  it 
is  acceptable. 

Interviews  provide  important  context  for  other  types  of  data  by  revealing  what  an 
operator  was  thinking  at  the  time  of  a  particular  activity.  We  can  ask  questions  whose 
answers  would  be  impossible  to  infer  from  direct  observation  or  chat  logs,  such  as  “What 
prompted  you  to  ask  for  that  information  at  that  point?”  “How  did  you  know  to  share  the 
information  with  those  particular  individuals?”  “How  did  you  reconcile  those  conflicting 
pieces  of  information?”  “Can  you  tell  us  why  you  double-checked  that  particular 
source?”  Unfortunately,  interview  responses  are  by  definition  subjective.  They  are 
filtered  through  an  individual’s  frame  of  reference,  cognitive  processing,  and  memory. 
Interview  data  can  suffer  from  hindsight  bias  or  rationalizations,  as  well  as  latency  issues 
when  questions  are  asked  in  retrospect.  It  is  therefore  important  to  combine  interview 
data  with  other  sources. 

Chat  Log  Analysis 

In  conducting  chat  analyses,  we  log  and  comb  through  the  text  of  instant 
messages  sent  from  an  individual  to  other  individuals  or  to  the  members  of  “chat  rooms”. 
These  rooms  are  organized  along  convenient  topics  and  subgroups  such  as  time-sensitive 
targeting,  weapons,  intelligence,  or  internal  coordination.  Each  operator  joins  the  subset 
of  chat  rooms  he  deems  most  useful  for  his  individual  tasks,  and  for  his  information 
sharing  needs  with  others  on  the  team. 

Data  from  chat  logs  have  the  advantages  of  immediacy  and  objectivity;  messages 
are  captured  word  for  word  as  they  occur  in  real  time  so  there  is  no  bias  or  time  lag 
between  what  occurred  and  what  is  recorded.  The  clear  documentation  of  senders, 
recipients,  information  content,  and  time  stamps  allows  for  the  re-creation  of 
collaborative  threads  across  multiple  participants  from  beginning  to  end.  Unfortunately, 
chat  logs  do  not  reflect  the  other  modes  of  communication  that  were  occurring 
simultaneously.  Compared  to  interview  data  or  observational  data,  chat  logs  offer  a 
limited  view  into  operator  motivations  and  the  environmental  context  surrounding  a 
discussion  thread. 

Fortunately,  we  have  found  that  data  from  field  observation,  interviews,  and  text 
messages  used  in  concert  create  a  powerful  mechanism  for  understanding  collaboration. 
For  example,  suppose  we  note  an  interesting  field  observation  such  as  two  people  in 
heated  discussion  over  a  target,  gesturing  at  a  display.  We  would  follow  up  by  asking 
clarifying  interview  questions  during  a  break  in  the  action,  such  as  “Can  you  tell  us  more 
about  your  discussion  with  operator  x  at  time  t?”  “Was  there  a  difference  of 
interpretation?”  “Was  the  information  you  needed  available?”  We  would  also  check  the 
relevant  chat  logs  surrounding  that  time  to  see  what  those  two  people  were  trying  to 


understand  or  decide.  As  shown  in  Figure  1,  the  combination  of  observation,  chat  and 
interviews  -  in  essence,  people’s  actions,  words,  and  thoughts,  respectively  -  helps  us 
capture  the  intersection  between  the  teams  of  operators,  their  available  systems,  and  their 
complex  environment. 


FINDINGS 

We  collected  field  data  from  numerous  joint  exercises  and  experiments  in  2004. 
The  commonality  was  an  emphasis  on  the  team  of  operators  engaged  in  Time- Sensitive 
Targeting  decisions.  In  these  scenarios,  an  unplanned  target  emerges  and  the  team  must 
locate  and  track  it,  redirect  intelligence,  surveillance  and  reconnaissance  assets  in  order  to 
determine  the  exact  nature  and  importance  of  the  target,  and  potentially  identify  and 
redirect  an  appropriate  weapons  platform  to  engage  the  target  -  all  within  a  short  time 
window.  These  teams  must  typically  work  across  service  boundaries,  or  even  across 
coalition  nation  boundaries,  in  order  to  effectively  cover  an  area  of  responsibility  and 
bring  all  necessary  assets  to  bear. 

It  is  common  to  think  of  human  operators  as  having  primarily  an  “information 
processing”  role  -  gathering,  transmitting  and  analyzing  information  much  as  computer 
systems  do  (albeit  with  greater  susceptibility  to  error  and  biases).  While  information 
processing  is  certainly  a  key  piece  of  the  human  role,  our  research  has  highlighted  several 
more  complex  and  subtle  roles  that  humans  play.  These  functions  are  vital  for  knitting  all 
the  pieces  of  information  together  in  context,  and  typically  require  the  ability  to  make 
multi-faceted  judgments,  to  handle  ambiguity,  and  to  be  adaptive.  Below,  we  describe 
some  of  the  critical  functions  that  operator  teams  perform.  They: 

•  validate  information,  and  determine  where  to  get  more  if  needed 

•  engage  in  collaborative  sensemaking  -  handling  ambiguous  or 
conflicting  information 

•  establish  trust  and  credibility  with  one  another 

•  maintain  sufficient  team  awareness  to  enable  effective  coordination, 
even  when  not  co-located 

•  judge  who  should  -  and  should  not  -  receive  information,  balancing  the 
need  for  sharing  against  the  danger  of  cognitive  overload 

•  select  appropriate  communication  modalities  for  sharing  information  of 
varying  importance,  time-sensitivity,  and  intended  audience 

•  cue  other  team  members  to  important  information,  emerging  events,  or 
changing  priorities. 

Determine  trustworthiness  and  reliability  of  information,  and  where  to  get  more 

We  observed  many  instances  where  human  operators  served  the  critical  function 
of  vetting  information  to  determine  its  trustworthiness  and  reliability  before  acting  on  it. 
This  is  particularly  important  in  the  kind  of  decision  making  context  studied  here  in 
which  ambiguities  regarding  enemy  targets,  rules  of  engagement,  and  priorities  abound. 
As  one  interviewee  commented,  “It  is  important  to  sort  out  what  information  really 
matters  and  to  verify  the  source  of  the  information  before  acting  on  it.” 


For  instance,  people  sometimes  double-checked  information  that  was  surprising  in 
context.  In  one  instance,  a  Lieutenant  was  surprised  to  see  an  order  for  an  Unmanned 
Aerial  Vehicle  (UAV)  to  return  to  base  because  it  was  just  about  to  provide  video  for 
Battle  Damage  Assessment  (BDA)  on  a  target.  (“RTB2??  Just  when  we  were  seeing 
impacts?”)  This  type  of  proactive  human  verification  avoided  potential  errors  or 
misunderstandings.  It  is  a  difficult  function  to  automate  because  it  requires  an 
understanding  of  the  broader  context,  namely  that  in  this  case  there  was  an  imminent 
expectation  of  a  desired  result  (video  to  indicate  whether  the  target  had  actually  been 
destroyed)  and  the  return  to  base  request,  if  correct,  would  prevent  that  outcome. 

In  other  cases,  people  requested  verification  of  the  source  of  the  information 
before  they  determined  how  to  respond.  In  the  case  of  the  surprising  RTB  order  noted  by 
the  Lieutenant,  her  Lieutenant  Colonel  wanted  to  know  the  identity  of  the  person  who 
gave  the  order  via  chat  (“Who  is  SAC  [chat  identifier]?”).  In  other  cases,  we  observed 
operators  requesting  a  seconding  of  important  information  in  order  to  establish  it  as 
trustworthy  {Have  you  guys  seen  any  intel3  on  this  target?).  Their  experience  and 
personal  relationships  with  others  on  the  team  informed  these  judgments  (“In  the  past,  he 
has  provided  very  reliable  intel.”). 

When  operators  saw  information  that  was  surprising,  confusing  or  incomplete, 
they  generally  took  the  initiative  to  get  clarification.  An  important  component  of  this 
information  verification  and  validation  process  was  knowing  where  to  go  for  the 
additional  information.  Although  we  might  expect  or  hope  that  the  sources  of  all  types  of 
information  would  be  predetermined  and  known,  this  is  not  typically  the  case.  Because 
there  is  so  much  information  coming  from  so  many  sources  in  so  many  forms,  and 
because  it  may  be  processed  or  updated  by  many  systems  and  people,  it  is  often 
challenging  to  find  the  needed  information  under  time  pressure.  As  some  operators 
explained  when  interviewed,  “It  takes  experience  to  know  where  to  get  the  right 
information.”  “It’s  usually  there,  but  knowing  where  to  look  is  key.” 

As  a  result,  the  power  of  the  informal  social  networks  that  teams  created  became 
apparent.  Operators  called  or  emailed  one  another  for  guidance;  teams  with  experience 
working  together  typically  knew  “who  to  call”  to  get  the  needed  information,  or  at  least  a 
pointer  to  it.  With  less  experienced  teams  or  more  novel  situations,  people  used  chat  to 
“information  shop,”  in  which  case  they  broadcast  a  request  for  the  desired  information 
{Does  anyone  know  where  I  can  find  out  about...  ?). 

Despite  operator  creativity  and  social  networks,  we  observed  instances  in  which 
the  desired  information  did  exist  but  operators  did  not  find  out  about  it  until  after  the  fact. 
(“I  knew  there  was  probably  a  file  on  this,  but  I  couldn’t  find  it  in  time.”)  There  are 
many  opportunities  for  improved  information  management;  the  information  must  not 
only  exist,  but  be  transparently  available  and  easily  accessible. 

Engage  in  collaborative  sensemaking 

As  previously  described,  people  often  play  an  important  role  in  verifying 
information  before  acting  on  it.  This  becomes  even  more  challenging  when,  despite 
verifying  its  source  and  provenance,  the  information  remains  fundamentally  ambiguous 
or  even  conflicting.  In  these  cases,  the  team  must  engage  in  collaborative  sensemaking. 


2  RTB  =  Return  To  Base 

3  Intel  =  intelligence 


What  does  the  information  really  mean?  What  are  the  implications  within  the  current 
context?  What  effects  might  be  anticipated? 

For  instance,  we  observed  a  team  working  with  a  UAV  in  the  vicinity  of  two 
airfields.  The  information  coming  from  the  UAV’s  video  was  deemed  valid  and  reliable, 
but  didn’t  make  sense  to  some  members  of  the  team  (“Is  that  the  wrong  airstrip? 
NorthTAC  is  at  the  base  of  a  ridge  and  I  don’t  see  any  of  the  ridge  in  the  video.”).  The 
team  struggled  to  interpret  the  information  and  its  implications  in  context;  did  it  mean  the 
UAV  was  flying  over  the  wrong  airfield,  as  some  thought,  or  that  it  was  directly  over  the 
expected  ridge  making  it  impossible  to  see? 

In  other  cases,  the  rules  of  engagement  did  not  unambiguously  specify  what  did 
and  did  not  qualify  as  a  dynamic  target,  and  whether  it  was  acceptable  to  engage  a 
particular  target.  In  one  instance,  a  lively  back  and  forth  discussion  ensued  via  chat, 
phone,  and  face-to-face  before  the  commander  made  a  judgment  call.  As  one  operator 
put  it,  “Rubbish  in,  rubbish  out.  You’ve  got  to  have  a  human  in  the  loop  when  there’s 
ambiguity.” 

Establish  trust  and  credibility 

Another  function  human  operators  perform  is  the  establishment  of  trust  and 
credibility  between  members  of  the  decision  making  team.  This  plays  a  critical  role  in 
their  ability  to  collectively  validate  incoming  information,  and  to  effectively  resolve 
differences  of  interpretation  when  information  conflicts  or  is  unclear.  In  time  pressured 
scenarios,  the  absence  or  presence  of  interpersonal  trust  and  credibility  can  be  the 
determining  factor  in  which  personal  opinions  and  supporting  system  information  get  full 
consideration. 

We  observed  a  team  interchange  in  which  this  mutual  trust  was  lacking.  A  lower 
ranking  Master  Sergeant  tried  to  alert  others  on  the  team  that  a  UAV  was  over  the  wrong 
airfield,  with  very  little  time  remaining  to  take  corrective  action.  He  pointed  out  the 
expected  but  missing  ridge  in  the  UAV  video  and  convinced  his  Lieutenant  that  his 
interpretation  was  correct;  she  had  a  Staff  Sergeant  radio  this  message  to  the  Major  in 
charge  at  a  nearby  but  separate  location.  The  Staff  Sergeant  was  rebuffed,  and  reported 
that  “they  are  too  busy  to  listen  right  now.”  The  Lieutenant  then  tried  to  deliver  the 
message  herself  face  to  face,  but  was  also  unsuccessful.  Under  severe  time  pressure,  the 
Master  Sergeant  and  Lieutenant’s  perception  of  the  situation  was  discounted  as  less 
credible  than  others,  and  so  was  the  system  information  they  referenced  in  support  of 
their  position. 

It  takes  significantly  more  time  and  effort  to  establish  trust  and  cohesiveness 
across  organizational  boundaries,  since  corresponding  team  members  will  typically  not 
be  co-located,  not  know  each  other  well,  and  not  share  all  of  the  same  goals,  values, 
priorities,  procedures,  and  social  or  cultural  conventions.  During  one  experiment  we 
attended,  a  potential  target  being  tracked  would  have  required  sharing  of  assets  between 
the  Air  Force  and  Army  should  it  be  engaged.  The  Air  and  Land  Component 
Commanders  were  geographically  separated.  As  the  Time- Sensitive  Targeting  team 
followed  this  potential  target,  we  observed  their  team  leader  monitoring  the  Land 
Component  Commander’s  status  on  a  display  of  coordinated  activities.  He  proactively 
contacted  his  Army  counterparts  the  several  times  via  phone  and  chat  although  no 
immediate  actions  were  needed.  When  interviewed  about  these  communications,  he 


explained  that  he  wanted  to  “get  them  energized”  and  “ensure  buy-in”  and  coordination 
before  the  potential  target  in  question  became  a  Time- Sensitive  Target.  He  recognized 
that  shared  situation  awareness  and  a  relationship  of  trust  were  critical  and  couldn’t  be 
immediately  established  at  decision  time. 

Maintain  sufficient  team  awareness 

In  order  for  operators  to  effectively  share  information,  coordinate  activities,  and 
create  shared  situation  awareness,  they  must  maintain  “ team  awareness.”  This  involves 
knowing  the  identities  of  others  on  the  team,  the  activities  they  are  engaged  in,  what  they 
know  and  do  not  know  at  a  given  time,  whether  they  are  present  at  their  stations  at  a 
given  time,  and  how  busy  they  are.  Without  this  awareness,  they  may  share  information 
inadequately  or  excessively,  misinterpret  a  response  or  a  delay  in  receiving  one,  and  lose 
trust  or  shared  situation  awareness. 

During  one  event,  two  team  members  were  observed  having  an  animated  face-to- 
face  discussion  regarding  a  target  they  had  not  engaged  in  time.  There  was  confusion  as 
to  whether  it  had  been  missed  inadvertently,  or  deliberately,  and  the  rationale.  (“Did  we 
know  about  these  SCUDS?”  “Yes,  but  ROE4  didn’t  let  us  attack.”  “Oh,  we  didn’t  know 
you  knew!”)  In  another  instance,  a  team  leader  expressed  the  importance  of  awareness  of 
team  members’  workload  and  cognitive  load  when  coordinating  on  tasks:  “I  need  to 
know  if  other  people  on  my  team  are  overwhelmed.  Are  they?  I  don’t  know.” 

Maintaining  team  awareness  is  particularly  difficult  when  the  team  is  not  co¬ 
located  and  cannot  readily  see  or  hear  one  another.  In  these  environments,  the  team  must 
use  a  mix  of  technologies,  such  as  tailored  coordination  displays,  phone  calls,  and  instant 
messages,  to  keep  abreast  of  others’  activities.  No  single  technology  currently  addresses 
the  team  awareness  issue. 


OPERATOR  ADAPTATIONS  OF  TECHNOLOGY  OR  PROCESSES 

Researchers  have  noted  that  although  systems  are  typically  designed  with  specific 
uses  and  purposes  in  mind,  teams  will  appropriate  those  tools  in  whatever  way  best  suits 
their  goals  and  needs. 

People  are  “purposive,  knowledgeable,  adaptive,  and  inventive  agents  who 
engage  with  technology  in  a  multiplicity  of  ways  to  accomplish  various  and 
dynamic  ends.  When  the  technology  does  not  help  them  achieve  those  ends,  they 
abandon  it,  or  work  around  it,  or  change  it,  or  think  about  changing  their  ends  ” 
(Orlikowski  2000,  p.423). 

Indeed,  some  of  the  most  interesting  lessons  learned  from  observing  technology- 
supported  teams  in  situ  involved  ways  people  adapted  either  their  processes,  or  their  use 
of  technology  to  meet  demands  in  real  time. 


4  ROE  =  Rules  of  Engagement 


Attention  Management 

It  was  common  for  TST  operators  to  have  two  computer  displays  each,  with 
multiple  application  windows  open  on  each  window.  The  left  screen  typically  contained 
an  application  for  coordinating  tasks  across  the  team,  as  well  as  a  map-oriented 
application  displaying  relevant  tracks  and  targets.  The  right  screen  was  commonly 
reserved  for  communication  and  typically  contained  half  a  dozen  or  more  chat  windows. 
Although  the  chat  tool  was  designed  to  alert  people  to  incoming  messages  by  providing 
an  auditory  “ping,”  this  became  ineffective  in  an  environment  with  many  chat  windows, 
competing  applications  on  the  other  screen,  and  constant  interruptions  from  others. 
Operators  were  clearly  struggling  to  manage  their  attention  and  maintain  situation 
awareness  when  simultaneously  bombarded  with  information  from  numerous  chat 
windows  and  via  various  other  modalities. 

Some  adaptive  behavior  involved  chat  in  particular.  In  order  to  sort  out  what 
information  among  all  the  chat  windows  was  new  and  what  had  already  been  read,  one 
operator  repeatedly  used  his  cursor  to  manually  highlight  the  bottom  line  in  each  chat 
window.  Thereafter,  any  time  a  new  message  arrived  it  would  replace  the  bottom  line 
and  the  highlighting  in  that  window  would  disappear;  the  chat  room  containing  the  new 
information  became  easy  to  identify  at  a  glance  as  the  one  without  highlighted  text.  Even 
if  the  operator  had  been  looking  at  his  map  display  or  talking  to  someone  when  new  chat 
came  in,  he  could  now  quickly  orient  himself  and  identify  the  information  needing  his 
attention.  Unfortunately,  even  this  clever  adaptation  broke  down  if  many  new  messages 
arrived  simultaneously,  or  if  the  operator  had  to  step  away  from  his  station  for  any  length 
of  time.  The  adaptive  behavior  met  a  short  term  need  on  a  limited  basis,  but  did  not  scale. 
It  alerted  us  to  a  larger  issue:  operators  are  cognitively  overloaded  and  require  improved 
technologies  for  filtering  information  and  directing  their  attention. 

Cue  other  team  members 

The  difficulty  in  keeping  up  with  the  flood  of  new  information  resulted  in  an 
adaptation  of  a  process,  as  well  as  the  previous  adaptation  of  a  technology.  We  observed 
operators  self-organizing  to  voluntarily  cue  one  another  to  important  developments  or 
changes  that  others  might  have  missed.  Team  members  took  the  initiative  to  note  when 
others  had  either  been  away  from  their  stations  or  otherwise  preoccupied  during  an 
important  event.  When  the  team  member  returned,  someone  would  inform  him  that  an 
event  or  item  of  potential  importance  had  been  missed,  and  where  to  look  to  quickly  get 
reoriented  (“You  missed  some  information  on  [target  x]  in  [chat  room  y].”) 

In  addition  to  ensuring  that  information  content  itself  was  not  missed,  this 
voluntary  cueing  propagated  meta-level  information  such  as  the  priority  of  the 
information.  The  fact  that  a  neighbor  expended  the  time  and  effort  to  cue  another  to  a 
piece  of  information  implicitly  communicated  that  the  information  was  deemed 
important.  Clearly  the  ability  to  cue  others  effectively  requires  sufficient  team  awareness, 
as  previously  described:  the  operator  must  understand  that  the  information  would  have 
been  relevant  to  the  other  team  member,  that  he  had  temporarily  been  absent  or 
overloaded,  and  that  he  was  now  available  and  capable  of  receiving  it. 


Judge  who  should  -  and  should  not  -  receive  certain  information 

Information  technology  facilitates  sharing  more  information  more  easily  with 
more  people,  yet  wider  information  sharing  was  not  always  beneficial  given  the  cognitive 
demands  of  the  operational  environment.  Since  supporting  technologies  sometimes  left 
operators  with  an  “all  or  nothing”  choice  for  information  sharing,  people  developed 
strategies  on  the  fly  to  balance  the  need  for  enough  information  sharing  against  the 
danger  of  excessive  distractions  or  cognitive  overload. 

For  instance,  the  TST  team  relied  on  an  application  that  presented  target 
information  in  a  matrix  format:  each  potential  target  appeared  in  a  row,  with  associated 
columns  and  tabs  for  entering  specific  status,  location,  and  intelligence  updates.  The  cell 
chief  used  this  application  to  keep  tabs  on  all  types  of  targets,  including  still  unverified  or 
“in  the  background”  targets  that  might  not  require  any  action  for  some  time.  The 
accepted  procedure  was  for  him  to  project  the  target  list  from  his  screen  onto  a  large  wall 
display  for  use  by  the  rest  of  the  team. 

While  it  was  desirable  to  fully  share  information  across  the  team  on  most  targets, 
the  cell  chief  found  unverified  “background”  targets  were  a  potential  source  of 
misdirection  of  the  team’s  limited  time,  energy,  and  resources.  Presented  with  an 
unsatisfactory  binary  choice  between  sharing  his  display  fully  or  not  sharing  it  at  all,  he 
adapted  by  projecting  his  display  on  the  data  wall  but  manually  hiding  the  rows 
corresponding  to  the  unverified  targets.  This  required  him  to  periodically  unhide  one  of 
these  rows  to  update  the  associated  information,  then  quickly  hide  it  again.  This 
workaround  met  short  term  needs,  but  at  the  expense  of  extra  time,  effort  and  attention 
for  the  operator.  Moreover,  the  approach  quickly  became  impractical  as  the  number  of 
potential  targets  being  considered  increased.  This  interplay  between  operators  and 
technology  indicates  an  underlying  need  for  a  collaboration  capability  supporting  more 
selective  information  sharing. 

Select  appropriate  communication  modalities  for  sharing  information 

We  observed  that  operators  not  only  had  to  determine  what  information  to  focus 
their  attention  on  and  with  whom  to  share  it,  but  also  had  to  make  countless  secondary 
judgments  about  how  to  do  so:  what  means  of  communicating  will  get  the  desired 
response  most  effectively? 

For  example,  an  operator  could  post  information  to  a  designated  database  or 
shared  application,  in  which  case  other  team  members  know  where  to  “pull”  the 
information  from  if  needed.  Another  option  was  to  “push”  the  information  via  face  to 
face  discussion  or  by  phone,  which  guaranteed  that  the  intended  audience  received  the 
information  but  provided  no  documentation  and  did  not  scale  to  more  than  a  few  people. 
Instant  messaging  was  another  possible  means  of  information  sharing,  in  which  case 
there  were  decisions  to  be  made  about  the  appropriate  chat  room(s)  to  address,  or  whether 
a  chat  in  private  directed  at  a  particular  individual  would  be  more  appropriate.  Still 
another  option  was  to  use  audio  chat  via  the  headsets,  in  which  case  the  operator’s 
comments  would  be  immediately  heard  by  the  selected  individual(s),  if  present. 

Teams  employed  these  various  modalities  differently,  depending  on  the 
importance  and  time- sensitivity  of  their  communication  and  the  intended  audience. 
Posting  target  information  to  the  appropriate  shared  application  was  part  of  the  planned 
process  and  effectively  accommodated  highly  structured  information  such  as  latitudes 


and  longitudes.  Sometimes  this  form  of  communication  was  supplemented  by  phone 
calls  or  text  messages  to  alert  others  that  new  information  was  available. 

Text  chat  was  commonly  used  when  a  communication  had  to  reach  many 
interested  parties  quickly,  and/or  when  it  was  judged  beneficial  to  have  a  written  record 
of  the  information  for  later  reference.  However,  since  operators  were  typically  logged 
into  six  to  twelve  chat  rooms  each,  and  some  of  these  rooms  had  large  numbers  of 
participants,  the  amount  of  information  in  chat  rooms  could  become  overwhelming  and 
unfocused.  As  a  result,  many  private  chats  between  pairs  of  individuals  spun  off  from  the 
main  chat  rooms  so  that  two  people  could  expand  on  a  thread  without  cluttering  the  main 
room  {Request  CIP5  with  you). 

When  information  was  particularly  important  or  time  critical  and  immediate 
response  was  needed,  operators  (especially  those  of  higher  rank)  often  chose  to  audio 
chat  to  a  subgroup.  (“Heads  up.  There  are  two  actions  heading  our  way,  take  a  look.”) 
Those  in  the  selected  chat  room  would  hear  the  spoken  communication  immediately 
through  their  headsets  and  redirect  their  attention.  Unfortunately,  if  their  headsets  were 
off  or  they  were  away  from  their  stations,  they  missed  the  information  and  had  no  way  of 
even  knowing  they  had  missed  it. 

Face-to-face  conversations  were  preferred  when  people  were  trying  to  convince  or 
influence  another,  when  they  were  having  difficulty  resolving  something  ambiguous,  or 
when  a  topic  was  difficult  to  explain.  Unfortunately,  as  already  discussed,  face-to-face  is 
time-consuming,  distracts  people  from  the  other  modalities,  and  is  not  possible  for 
distributed  teams. 

Machine  to  Machine  and  Loss  of  Context 

Although  technology  such  as  machine  to  machine  transfer  is  invaluable  for 
speeding  transmission  and  avoiding  errors  from  manual  entry,  there  are  complexities  to 
address  involving  the  intersection  of  technology  and  human  decision  making  processes. 
For  instance,  when  operators  received  data  via  machine  to  machine  transfer  rather  than 
from  another  member  of  the  team,  they  could  lose  the  ability  to  infer  important 
contextual  information  such  as  the  data’s  source,  pedigree,  and  why  it  was  being 
provided  at  that  time.  As  a  consequence,  operators  sometimes  found  it  necessary  to 
reconstruct  some  of  that  meta-information  in  order  to  judge  the  information’s  validity, 
provenance,  and  relevance,  and  determine  how  to  proceed. 

We  also  observed  operator  frustration  when  they  saw  automatically  generated 
inputs  overwrite  their  manual  inputs  to  a  system;  in  one  instance  the  rows  were  being 
reordered  as  an  operator  tried  to  select  and  edit  one  on  his  screen.  In  another  instance, 
machine  to  machine  inputs  overwrote  priorities  an  operator  had  recently  entered.  During 
an  interview,  one  individual  commented,  “Operators  wanted  machine  to  machine,  but  got 
more  than  they  asked  for:  human  decision-makers  were  not  always  in  the  loop. 

Sometimes  retasking  started  without  them. . .”  Clearly  new  technologies  will  disrupt 
familiar  decision  making  processes  —  hopefully  en  route  to  making  them  more  efficient  — 
but  we  must  plan  and  train  for  these  changes. 


5  CIP  =  Chat  in  Private 


CONCLUSION 


In  Cognition  In  The  Wild,  Hutchins  talks  about  the  “cognitive  functional  system” 
composed  of  both  humans  and  technology. 

“  ...the  computational  power  of  the  system  composed  of  person  and 
technology  is  not  determined  primarily  by  the  information-processing 
capacity  that  is  internal  to  the  technological  device,  but  by  the  role  the 
technology  plays  in  the  composition  of  a  cognitive  functional  system.  ” 

(Hutchins,  1995,  pl55) 

This  paper  described  some  of  the  contributions  that  humans  make  to  this 
“cognitive  functional  system.”  It  also  documents  some  of  the  challenges  inherent  in 
making  the  collaboration  of  humans  and  technology,  as  well  as  the  collaboration  with  one 
another,  as  effective  as  possible  in  complex  environments.  As  Brown  and  Duguid  aptly 
note,  “Some  futurists  seem  continuously  anxious  to  replace  humans. .  .in  certain  tasks 
without  quite  appreciating  how  people  accomplish  those  tasks.  In  general,  it  will  be 
better  to  pursue  not  substitution  but  complementarity. . .  [which]  requires  seeing  the 
differences  between  information-processing  agents  and  human  agency”  (p.62). 


ACRONYMS 


BDA  Battle  Damage  Assessment 

CIP  Chat  in  Private 

ROE  Rules  of  Engagement 

RTB  Return  To  Base 

TST  Time-Sensitive  Targeting 

UAV  Unmanned/Unpiloted  Aerial  Vehicle 
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Problem 


•  Greater  complexity, 
increased  reliance 
on  teams 

•  Human  decision 
making  is  often  the 
‘long  pole  in  the  tent’ 
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Impact 


“Some  futurists  seem  continuously  anxious  to  replace  humans... 

In  general,  it  will  be  better  to  pursue  not  substitution  but 
complementarity...  [which]  requires  seeing  the  differences 

between  information-processing  agents  and  human  agency.” 

From  Brown  and  Duguid  (2000),  The  Social  Life  of  Information. 
Boston,  MA:  Harvard  Business  School  Press. 


•  Understand  “human  agency”  to  enable  complementarity  between 
humans  and  information  technology 


•  Human  adaptations,  emergent  behavior 

-  can  reveal  unmet  needs  for  effective  collaboration  in  complex 
environments 
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Collaboration:  Challenges  for  Data 
Collection,  Measurement  &  Analysis 


Goal:  Study  Human  Teams  with  Technologies  in  Rich  Environment 


Setting  the  Stage: 

A  Glimpse  of  Team  with  Systems  in  Context 


Example  of  2  minutes  with  Cell  Chief: 

•  Text  chats  in  private  to  an  individual  regarding  a  possible  leadership  target 

•  Audio  chats  with  someone  else  to  ask  why  they’re  working  a  target 

•  Starts  another  audio  conversation 

•  Accepts  a  text  chat  message  requesting  chat  in  private 

•  Accepts  another  private  chat  going  about  priorities  (“Is  this  urgent?”) 

•  Uses  audio  chat  to  give  entire  team  a  heads  up  (“There  are  two  actions  heading 
our  way,  take  a  look.”) 

•  At  the  moment,  he  has  three  private  chats  going,  plus  the  8  regular  chat 
windows,  then  audio  chat  comes  in  for  him... 

•  Initiates  a  follow-up  audio  chat  to  another  individual.  Gets  no  reply,  so  text  chats 
in  private:  “Call  me  on  the  headset  or  visit  me..." 

•  Meanwhile,  gets  a  different  kind  of  text  message  from  left  screen  application... 

[Note:  This  is  all  separate  from  target  list  application,  email  messages, 
face-to-face  conversations,  public  announcements,  wall  displays...] 
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Attention  Management: 

“Where  do  I  focus  my  attention?” 


•  Operator  juggled  8+  chat  windows  on  right  screen  alone: 

Hears  “ping”  for  an  incoming  chat  message,  “But  which  room  is  it  in?” 

-  Adaptation:  highlights  most  recent  message  in  each  window;  new 
messages  appear  unhighlighted,  easy  to  spot  (but  must  repeat...) 


•  Self-orqanize  to  cue  others: 

-  To  returning  TST  Chief: 

“You  missed  lots  of  chat  on...” 
(indicates  target  and  chat  room) 


Technology  could  better  support  human  focus 


5 


MITRE 


Information  Sharing: 

“Who  else  should  see  this?” 


•  Chief  audio  chats  regarding 
leadership  target 

“Why  are  you  working  it?  Stop  - 
it’s  a  misdirect.  Target  is  ‘in  the 

background’...  Don’t  work  it 
until  it’s  pushed  to  you.” 


•  Adaptation  to  avoid  misdirecting  others 

TST  Chief’s  left  screen  displays  target  information  on  AOC  data  wall 
-  Until  sure  of  ID,  he  enters  some  targets  as  hidden  rows 


More  information  sharing  isn’t  always  desired 
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Managing  Relationships  for 
Speed  of  Command 


Teams  of  Teams 

•  In  one  scenario,  Air  and  Land  Component  Commanders  coordination  key 

•  TST  Chief  proactively  monitored  LCC’s  indicator  on  stoplight  chart  and 
communicated  with  them  even  when  no  action  needed.  Reported 
wanting  to  “get  them  energized,”  ensure  buy-in  and  coordination  “before 
it’s  a  TST” 


Challenge  increases  if  crossing  organizational  boundaries 
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Observation  Highlights  from 
Time-Sensitive  Command  &  Control  Events 


Teams  Manage  Complex,  Fluid 
Environment 
manage  attention 
cue  others 
handle  ambiguity 

Teams  Manage  Information 

determine  reliability  of  information 
determine  who  should  see  what,  and  how 

Teams  Manage  Team  Dynamics 


coordinate  roles,  priorities 
establish  trust,  credibility,  buy-in 
maintain  team  awareness 
develop  shared  situation  awareness 


Teams  T 

Manage 

More  Thar 

Information 
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